fix(tern): settle sequential tasks when the engine loses in-flight work by aparajon · Pull Request #1113 · block/schemabot

aparajon · 2026-08-23T04:24:10Z

What breaks today

The loop driving one table's schema change polls the engine until it reports the work finished or failed. That is the loop's only exit — so if the engine forgets the work, there is no exit.

Engines do forget. Their view of in-flight work lives in one process's memory, and it vanishes when that process restarts, or when a sibling drive on the same pod drains the engine in the window between the work finishing and the next poll. From then on the engine answers "no active schema change", with no error, forever — which reads exactly like a stale poll, and the loop is built to ride stale polls out.

                            engine forgets the work it accepted
                                            ╳
   poll ─────── poll ─────── poll ─────── poll ─────── poll ─────── …
  running      running      running     "nothing"    "nothing"
                                                          │
                              stored state still says "in flight",
                              so the drive waits it out. Forever.

Nothing notices, because from the outside nothing is wrong: heartbeats stay healthy, the error budget never burns, recovery never fires. Meanwhile the apply holds the database's one active-apply slot, so every later apply to that database queues behind work that will never finish.

The fix

When the engine reports no active work but stored state says the work is in flight, ask the target database — the one authority that cannot forget an outcome.

  what the target shows        the task settles as
  ──────────────────────────────────────────────────────────
  mid-revert (never read)  →   retryable
  change is present        →   completed
  change is missing        →   retryable, a fresh claim re-drives it

The catch is that "no active work" is overloaded. An engine that provisions after accepting work — cutting a branch, validating a deploy request — says the same thing about real, healthy work that simply hasn't started yet, and bouncing one of those re-drives an apply that was about to run fine. So engines declare how to read their own answer, through a new optional engine.SynchronousWorkRegistration. Spirit and the PostgreSQL engine declare it: both run the work in this process with nothing to provision, so once Apply has returned the work is either running or gone and the first such report is conclusive. Undeclared engines are assumed to provision and get a bounded wait first, which is the safe default.

That wait is a duration rather than a count of polls, because what it has to outlast is engine behaviour measured in wall-clock time; a poll count would silently shrink to nothing if someone shortened the poll interval.

The revert case is the sharp edge. After a forward change cuts over, the live schema matches the reviewed target by definition — so a match says nothing about whether the revert this task was driving ever finished, and completing on it would report a successful schema change while the revert is gone. A revert-phase task settles retryable without reading the target at all, the same guard the resume path already applies. Beyond that, verification can only ever land failed_retryable, never failed, and errors reading the target burn the existing consecutive-error budget so this path can't become a second silent loop.

Also: the drive says when it's stuck

The same loop gains a stall watchdog. When a task's state and progress counters haven't moved for the warning interval, it logs once per interval with the task's triage attributes, how long it's been motionless, and what the engine reports. It observes only — it never changes task state. Tasks parked at an operator gate (held cutover, deferred deploy, open revert window) are motionless by design and stay quiet, and since the watchdog still sees every poll, entering or leaving a gate restarts its clock.

Engine progress is a display feed; outcomes belong to durable state. This is the first drive path where the target database settles a task when the engine's in-memory view and stored state disagree — progress polls decide what operators see, durable state and the target decide what is true. The capability interface follows the precedent ExternallyAuthoritativeProgress set: instance-local memory is never trusted as truth unless an engine explicitly says it can be.

PR summary written by Claude Code (Opus 5).

Copilot

Pull request overview

This PR hardens the sequential Tern drive against “lost in-flight work” scenarios where the engine reports no active schema change while storage still shows a task running, which previously could lead to silent infinite polling and a wedged apply.

Changes:

Adds bounded “lost work” detection in the sequential poll loop and verifies target convergence via a re-plan when the pending/no-active signal persists.
Introduces a stall watchdog that rate-limits warnings when task state/progress does not move for a configured interval.
Expands sequential progress tests to cover lost-work settlement (converged vs not), stale-snapshot self-heal, and watchdog warning behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
pkg/tern/local_client.go	Adds LocalClient overrides for sequential poll cadence and stall-warning interval (primarily for tests).
pkg/tern/local_apply_sequential.go	Implements lost-work tolerance + verification, bounded error handling integration, and a stall watchdog with rate-limited warnings.
pkg/tern/local_apply_sequential_progress_test.go	Adds fixtures and tests for lost-work settlement paths and watchdog logging/rate limiting.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

The sequential drive's progress poll tolerates a short window of the engine reporting no active schema change for an in-flight task (a stale snapshot after an engine restart self-heals), then stops trusting the engine and verifies the target schema directly: a converged target completes the task through the normal completed flow, a target that still needs the change marks the task retryable so a fresh claim re-drives it, and verification errors count against the poll's bounded consecutive-error budget. Every branch logs with the task's triage attributes and the engine-reported state. The poll also carries a stall watchdog: a task sitting in the same state with unchanged progress fields for a full interval is warned about once per interval, without changing task state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…nd trust the engine on a clock A task in its revert phase is in flight, so the sequential drive's lost-work verification could reach it. Reading the target schema cannot settle a revert: the forward change has already cut over, so the live schema matches the reviewed target by definition and a match says nothing about whether the revert ever finished. That path reported the apply as a successful schema change while the revert it was undoing was gone. A revert-phase task is now marked retryable without consulting the target, matching the guard the resume path already applies for the same reason. The trust budget before that verification runs is now a duration rather than a count of polls. What it has to outlast is wall-clock engine behaviour, not a number of round trips: an engine that just restarted serves a stale snapshot until it catches up, and an engine whose remote work is still being provisioned or validated reports no active schema change for real, healthy work it has not begun executing. A poll-count budget silently shrinks to nothing whenever the poll cadence is shortened, which bounced a healthy apply to retryable and re-drove it — worse than waiting on an engine that provisions remote resources per attempt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… an operator gate A held cutover, a deferred deploy and an open revert window are all states the drive is meant to sit in without moving, because the next step belongs to an operator rather than to the engine. Progress fields do not advance there by design, so the stall warning fired once per interval for as long as the operator took to act — on every deferred cutover and deferred deploy. The watchdog still observes every poll, so entering or leaving one of these states restarts its clock and a genuinely stuck task still warns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The trust budget before the drive verifies the target schema was one constant for every engine, sized for the slowest of them. That is the wrong shape: how long a pending progress report stays ambiguous is a property of the engine, not of the drive. An engine that provisions after accepting work — cutting a branch, opening and validating a deploy request — reports pending for real, healthy work for as long as that setup takes, so a driver must give it time. An engine that publishes its tracked schema change before Apply returns has no such phase: once Apply has returned the work is either running or it is gone, so the first pending report about an in-flight task is already conclusive and there is nothing to wait out. A new optional engine.SynchronousWorkRegistration lets an engine declare which it is, and the sequential drive reads that declaration to size the budget. Spirit declares it: it runs in a goroutine of this process with nothing to provision, and Drain and Cancel are the only writers that clear the tracked state, both of which mean the work is not coming back. Engines that do not declare it keep the full budget, which is the safe default — provisioning is never mistaken for lost work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Kiran01bm · 2026-08-28T08:07:54Z

🤖 Review findings - created by Kiran's code review agent - for schemabot/pull/1113, 7882487.

Verdict: 7 findings — 3 non-blocking (sibling drive still hangs, untested settle-error branch, comment/constant drift), 4 general suggestions.

Non-blocking

1. The grouped drive has the identical unbounded-pending hang, and Spirit reaches it on every defer_cutover MySQL apply. pkg/tern/local_apply_grouped.go:571

usesGroupedApply routes every Vitess apply and every MySQL apply with defer_cutover to pollForCompletionAtomic:

return apply.DatabaseType == storage.DatabaseTypeMySQL && storage.ApplyOptionsFromMap(options).DeferCutover

(pkg/tern/local_apply.go:561)

handleAtomicProgressTick has no lost-work branch and no bound on consecutive pending reports — on a successful poll it does ps.consecutiveErrors = 0 and syncs task state, and taskStateWithNoBackwardProgress pins the pending report back to running (pkg/tern/local_client.go:3297), so the loop is exactly the shape the PR just fixed on the sequential side.

Failure scenario: a MySQL defer_cutover apply is driven by Spirit; Spirit's tracked change is cleared (Drain, or a re-keyed engine after restart), Progress returns StatePending forever (pkg/engine/spirit/spirit.go:717), and the grouped drive polls indefinitely holding the database's active-apply slot. Spirit is the one engine this PR declares synchronous, so the engine with the strongest evidence available gets none of the benefit on this path. Reasonable as a follow-up, not as scope creep here.

2. The failed-verification branch — including the bound the comment says "must never become an unbounded loop" — has no test. pkg/tern/local_apply_sequential.go:641

if consecutiveErrors >= maxConsecutiveProgressPollErrors {
    c.markTaskRetryable(ctx, task, fmt.Sprintf("engine reports no active schema change for an in-flight task and target verification failed after %d consecutive errors; see server logs", consecutiveErrors))

settleLostEngineWork can only return a non-nil error from three places (:829, :832, :836), and none of them is reachable from the new tests — so neither is this escape hatch that counts them. lostWorkEngine.Plan always returns (e.planResult, nil) and the fixture's plan store is &scriptedPlanStore{plan: ...} with no err (pkg/tern/local_apply_sequential_progress_test.go:299). scriptedPlanStore already carries an err field (pkg/tern/local_control_resume_test.go:343), so pinning this is a two-line addition. Failure scenario: a later refactor drops or weakens the bound and a task whose engine lost the work and whose target is unreadable re-plans forever, holding the active-apply slot — the exact failure this PR exists to remove — with nothing in CI to notice. Worth also covering the interleaving, since the shared budget means 9 transient poll errors plus 1 verification failure produces the message "target verification failed after 10 consecutive errors", which misattributes nine of them.

3. The new constant's doc comment asserts a coupling to the grouped poll that does not exist in code. pkg/tern/local_apply_sequential.go:454

// maxConsecutiveProgressPollErrors bounds how many consecutive progress
// poll failures the sequential drive tolerates before settling the task,
// matching the grouped poll.

The grouped poll keeps its own literals — ticker := time.NewTicker(500 * time.Millisecond) (:571) and if ps.consecutiveErrors >= 10 { (:649) — so tuning maxConsecutiveProgressPollErrors (say, lowering it because a wedged apply blocks the deploy queue) silently changes only one drive and falsifies the comment with no compiler or test signal. AGENTS.md's No fragile comments rule points the same way. Adopting both constants at the two grouped call sites completes the sweep.

General suggestions

4. The Postgres engine qualifies for SynchronousWorkRegistration but does not declare it. pkg/engine/postgres/apply.go:55

e.claimProgress(key, progressResult(engine.StateRunning, "preflight", started, change, ""))

runs synchronously before e.wg.Go(...) and before Apply returns, and Progress returns the pending sentinel only on a key mismatch (pkg/engine/postgres/apply.go:236) — i.e. exactly the "no post-acceptance provisioning phase" the new interface describes. Undeclared, it falls through to defaultLostEngineWorkPendingBudget, so a Postgres sequential apply whose progress was lost polls a dead task for two extra minutes. Correct-but-slow (the default is deliberately the safe one), but the one-method declaration alongside Spirit's would make the second in-process engine settle as fast.

5. Repeated verification failures re-plan the whole database once per 500 ms tick with no backoff. pkg/tern/local_apply_sequential.go:630

action, settleErr := c.settleLostEngineWork(ctx, apply, task, result.State)
if settleErr == nil {
    return action
}

On error the loop continues, and the next tick re-enters the exhausted branch and calls tableStillNeedsChange again — each call is a full planWithEngine over plan.SchemaFiles (pkg/tern/local_control_resume.go:443), i.e. a whole-database schema diff. That is up to ten full diffs in ~5 s against a target that is already degraded (which is why verification is failing). Bounded, so it cannot spin, but a short backoff on the verification retry — as distinct from the poll cadence — would avoid hammering a sick target.

6. The stall warning carries no throttle context, so an engine-paced copy reads as a wedged one. pkg/tern/local_apply_sequential.go:715

if stalledFor, warn := watchdog.observe(now, taskProgressSnapshotOf(task)); warn && !taskWaitsForOperatorAction(task.State) {

taskProgressSnapshot captures only state/rowsCopied/progressPercent/checksumRowsChecked (:867), and Task.LogAttrs() contains no throttle field (pkg/storage/logattrs.go:117) — yet the loop persists task.Throttled = tp.Throttled two blocks earlier (:673) from a flag Spirit sets when the copier is paced on replica lag (pkg/engine/spirit/spirit.go:878). The new test's own comment names "a throttled engine" as a legitimate reason to sit still and says the point of the warning is that "an operator reading the logs must be able to tell a slow task from a wedged one" — but the warning as emitted cannot. Log-only, no state change. Adding "throttled", task.Throttled, "throttle_reason", task.ThrottleReason to the warn attrs (cheaper than suppressing) makes it answer its own triage question.

7. Two verdicts now exist for the same physical condition. pkg/tern/local_apply.go:376

if result.Message == "No active schema change" {

The conflict check marks that task state.Task.Failed — permanently, with no target read — while the new drive path marks the same condition failed_retryable after proving the target still needs the change (pkg/tern/local_apply_sequential.go:851). An operator triaging "the engine forgot my work" gets a permanently failed task if a competing apply's conflict check noticed first and a retryable one if the drive noticed first. The older site also keys off the engine's message string rather than the new engineReportsLostWork predicate, which is exactly the kind of raw string comparison AGENTS.md's State Comparisons rule discourages. Pre-existing and out of scope here, but the new predicate is the natural thing for that site to adopt.

The one thing that could have broken, verified

Spirit declaring RegistersWorkSynchronously() == true (pkg/engine/spirit/spirit.go:299) gives it a zero trust budget, so a single pending progress report immediately terminalizes a task durable storage believes is in flight — completing a live production MySQL schema change or bouncing it to failed_retryable. Everything rests on Spirit never reporting pending for healthy work.

I proved it safe three ways:

Producer. Progress returns StatePending only under if e.runningSchemaChange == nil (spirit.go:717); no other branch produces it.
Mutators. Grepping every write to runningSchemaChange in non-test code shows exactly two nil-ers: Drain (spirit.go:318) and Cancel (control.go:147). Stop, Start (control.go:202), Volume's stop/reconfigure/restart cycle (control.go:405) and HaltForShutdown all leave it set.
Concurrency. pollTaskToCompletion has exactly one non-test caller — runEngineTask at local_apply_sequential.go:245, immediately after a successful Apply on the same goroutine. The two other Drain() sites are the resume paths (local_control_resume.go:351, :725), which run under a fresh lease claim. The one window I chased — an operator cancel nil-ing the tracked change mid-poll — is closed at the source: LocalClient.Cancel queues a durable control request rather than cancelling inline (local_control.go:356), and cancelOwnedApply is reached only from processPendingCancelControlRequest (local_control.go:1204) on the drive goroutine itself, which returns handled before eng.Progress is ever called that tick.

I also tried and failed to build a lease-loss variant of that race: a drive whose lease is lost or presumed lost has its context cancelled by applyHeartbeatFailureStopsDrive (local_apply_sequential.go:420) before a peer can reclaim and Drain, and the poll re-fetches the task each tick and exits on an externally-terminal state (:574). Verdict: safe as built.

Verified correct

pkg/tern/local_client.go:3297 — the mechanism does not self-defeat. if engineProgressRank < storedProgressRank { return storedTaskState } means a pending report never rewrites task.State to pending, so prevState stays running on the next tick and the trust budget genuinely accrues across polls instead of resetting every second poll.
pkg/tern/local_apply_sequential.go:787 — observePending boundary behaviour: budget <= 0 exhausts on the first report, a positive budget never exhausts on the first report and only at pendingFor >= budget, and a still-set since after a failed settle keeps reporting exhausted rather than restarting the clock.
pkg/tern/local_apply_sequential.go:645 and :655 — relocating consecutiveErrors = 0 and logEngineResumeOnce below the lost-work block is correct: the settle-error path continues before the reset so failed verifications accumulate, while the inside-budget path falls through and resets exactly as before.
pkg/tern/local_apply_sequential.go:820 — the revert-phase guard reads task.State before line 659 mutates it, so it sees the same prevState the caller tested; Reverting is in IsInFlightTaskState (pkg/state/task.go:80) so the guard is reachable.
The suspected revert_window gap is refuted: RevertWindow is produced only by the PlanetScale engine (pkg/engine/planetscale/planetscale.go:827), which getEngine returns only for DatabaseTypeVitess (pkg/tern/local_client.go:2753), and usesGroupedApply sends every Vitess apply to the grouped drive — so a sequential task can never sit in revert_window, and the RevertWindow arm of taskInRevertPhase at this call site is defensive rather than a live gap.
pkg/tern/local_apply_sequential.go:834 — tableStillNeedsChange keys by (namespace, shard, table) via replanShardTableDDL (local_control_resume.go:447), the same reconciliation the proven resume path uses, so a sharded task cannot be settled by another shard's or keyspace's diff; and the not-converged branch is markTaskRetryable, never markTaskFailed.
pkg/tern/local_apply_sequential.go:898 — taskStallWatchdog.observe arms on the first call, measures from when movement stopped rather than from the last warning, rate-limits to one warning per interval, and resets both clock and latch on any change; taskProgressSnapshot is an all-comparable struct so snap != w.last is a valid equality test.
pkg/engine/engine.go:186 — RegistersWorkSynchronously(nil) is safe (type assertion on a nil interface yields ok == false), so a nil engine falls back to the conservative default rather than panicking, and test doubles that only embed engine.Engine correctly do not satisfy the interface, preserving the 2-minute default for pre-existing tests.
pkg/tern/local_apply.go:478 — transitionTaskState assigns task.State in memory before persisting, so runEngineTask's post-poll switch task.State maps settleLostEngineWork's Completed → taskContinue and FailedRetryable → taskFailed regardless of the returned taskAction.
Logging follows AGENTS.md: the new lines build on task.LogAttrs() and append "apply_id", apply.ApplyIdentifier — which Task.LogAttrs() does not already emit (pkg/storage/logattrs.go:117) — so there are no duplicate slog keys, and no internal numeric row ID is logged.
CI is green on 7882487 across Unit, Integration, E2E MySQL, E2E Vitess, E2E gRPC, E2E K8s and LocalScale — direct evidence that Spirit's new zero budget does not misfire on healthy sequential applies.

This review was generated by Claude Code (claude-opus-5).

…ost-work verification The PostgreSQL engine claims its tracked progress under the engine mutex before Apply returns and only Drain clears it, so a pending report about in-flight work is conclusive; declaring engine.SynchronousWorkRegistration lets a drive verify the target immediately instead of spending a trust budget the engine does not need. The grouped poll now shares the sequential drive's poll-interval and error-budget constants, and the retryable message for an unverifiable target attributes the error count to progress polls and target verification together rather than to verification alone. A new test pins that failed verification reads are bounded by the shared error budget and rest the task retryable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A throttled task and a wedged one look identical to the stall watchdog: no state or progress movement past the warning interval. Adding the engine's throttle flag and reason to the warning answers the first triage question — is the engine deliberately holding back — from the log line alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

aparajon · 2026-08-28T08:47:27Z

🤖 Thanks — addressed at 0cd1acd + b8a9c9e, finding by finding:

Grouped drive shares the hang — agreed, and taking you up on the follow-up framing: the grouped tick needs its own settle path, since its tasks share one engine progress report but each table's target has to be verified on its own, and a revert-phase task in the group can never be settled by a schema read. Queued as follow-up work rather than grown into this PR.
Settle-error branch untested — fixed. A new test fails the plan-store read on every settle attempt and pins the whole branch: verification failures are counted by the same bounded budget, exhausting it rests the task retryable — never permanently failed, because nothing proved the target broken — and the engine re-plan is never reached when the plan row can't be loaded. The misattribution you spotted is also gone: the retryable message now credits the count to "progress polls and target verification" together instead of claiming verification failed N times when up to N−1 of those were poll errors.
Constant doc claims coupling that didn't exist — fixed by making the claim true rather than deleting it: the grouped poll now uses defaultTaskPollInterval and maxConsecutiveProgressPollErrors at both former literal sites, and the two constants' docs say they are shared by the sequential and grouped polls.
Postgres should declare synchronous registration — done. The engine claims its tracked progress under the mutex before Apply returns, and Drain — whose work is by definition not coming back — is the only writer that clears it, so the declaration is sound end to end; it gets the same test shape Spirit's declaration has. A Postgres drive now verifies the target on the first conclusive pending report instead of spending a two-minute budget the engine never needed.
Backoff between verification attempts — passed on this one deliberately. The settle call runs synchronously inside the tick loop, so attempts can never outpace the schema diff itself: the retry floor is the diff's own latency plus a full poll interval, and the whole phase is capped at the shared error budget before the task rests retryable. A second time knob would interact with both the poll cadence and the trust budget without changing that bounded outcome; happy to revisit if a real target shows the diff-paced floor is still too hot.
Throttle state on the stall warning — done, throttled and throttle_reason ride the warning, and the stall-watchdog test now asserts the attribute is present. It answers exactly the triage question you framed: whether a motionless task is the engine deliberately holding back.
Old conflict-check site — agreed it stays out of scope, and I'd add one reason beyond scope: that site keys on the verbatim idle-sentinel message (a cross-engine contract the Postgres engine documents explicitly) and settles to permanent Failed in a pre-apply conflict check, so adopting engineReportsLostWork there is a verdict change, not just a comparison cleanup. It belongs with the grouped-drive follow-up where lost-work semantics get unified.

Each change was mutation-verified: flipping the Postgres declaration to false, downgrading the exhausted-budget branch from retryable to failed, and dropping the throttle attributes each fail at least one test on re-run.

This reply was generated by Claude Code (Claude Fable 5).

# Conflicts: # pkg/engine/postgres/postgres_test.go

…t-dialect-classify * origin/main: fix(github): keep the PR progress comment updating between operation dispatch waves (#1104) fix(tern): classify materialized change DDL with the target dialect parser (#1187) fix(engine): resolve a cancel or stop that arrives before remote dispatch (#1184) fix: default connect and write timeouts on managed database connections (#1182) fix(storage): index the apply-operation claim ordering (#1180) fix(tern): generalize control resume state and complete cancels with no live engine work (#1179) fix(github): name each table's outcome in unsuccessful apply summaries (#1186) ci: peel tern and webhook into a dedicated integration shard (#1166) fix(engine): report a drained schema change's terminal outcome instead of pending (#1114) feat(serve): contain gRPC handler panics with recovery interceptors (#1164) feat(observability): tell operators when a log window hides older entries (#1185) fix(tern): settle sequential tasks when the engine loses in-flight work (#1113) fix(github): align PR comment severity glyphs with the shared vocabulary (#1135) fix(tern): release a database held by a stopped schema change (#1175) fix(plan): canonicalize drift DDL with the target's dialect parser (#1177) fix(e2e): stop injecting connection kills once the k8s pause is observed (#1178) # Conflicts: # pkg/webhook/templates/plan.go

…lassify' into kiran01bm/apply-comment-dialect * origin/kiran01bm/plan-comment-dialect-classify: fix(github): line-break non-MySQL DDL, schema labels for postgres fix(github): keep the PR progress comment updating between operation dispatch waves (#1104) fix(tern): classify materialized change DDL with the target dialect parser (#1187) fix(engine): resolve a cancel or stop that arrives before remote dispatch (#1184) fix: default connect and write timeouts on managed database connections (#1182) fix(storage): index the apply-operation claim ordering (#1180) fix(tern): generalize control resume state and complete cancels with no live engine work (#1179) fix(github): name each table's outcome in unsuccessful apply summaries (#1186) ci: peel tern and webhook into a dedicated integration shard (#1166) fix(engine): report a drained schema change's terminal outcome instead of pending (#1114) feat(serve): contain gRPC handler panics with recovery interceptors (#1164) feat(observability): tell operators when a log window hides older entries (#1185) fix(tern): settle sequential tasks when the engine loses in-flight work (#1113) fix(github): align PR comment severity glyphs with the shared vocabulary (#1135) fix(tern): release a database held by a stopped schema change (#1175) fix(plan): canonicalize drift DDL with the target's dialect parser (#1177) fix(e2e): stop injecting connection kills once the k8s pause is observed (#1178) # Conflicts: # pkg/webhook/templates/apply.go

Copilot AI lite review requested due to automatic review settings August 23, 2026 04:24

Copilot started reviewing on behalf of aparajon August 23, 2026 04:24 View session

Copilot AI reviewed Aug 23, 2026

View reviewed changes

Comment thread pkg/tern/local_apply_sequential.go

Comment thread pkg/tern/local_apply_sequential.go

aparajon force-pushed the armand/poll-lost-work branch from e950fc7 to 9f7610a Compare August 24, 2026 03:57

aparajon marked this pull request as ready for review August 24, 2026 04:27

aparajon requested review from JashLal, Kiran01bm, eeSeeGee, jayjanssen, jemiahw and morgo as code owners August 24, 2026 04:27

aparajon and others added 3 commits August 24, 2026 12:50

aparajon mentioned this pull request Aug 24, 2026

fix(engine): key PostgreSQL progress to every apply, not one slot #1130

Open

Kiran01bm approved these changes Aug 28, 2026

View reviewed changes

aparajon and others added 2 commits August 28, 2026 16:33

Merge remote-tracking branch 'origin/main' into armand/poll-lost-work

e466630

# Conflicts: # pkg/engine/postgres/postgres_test.go

aparajon merged commit 6e1417f into main Aug 28, 2026
34 checks passed

aparajon deleted the armand/poll-lost-work branch August 28, 2026 09:07

aparajon mentioned this pull request Aug 28, 2026

fix(tern): settle grouped applies when the engine loses in-flight work #1192

Open

Navigation Menu

Uh oh!

fix(tern): settle sequential tasks when the engine loses in-flight work - #1113

fix(tern): settle sequential tasks when the engine loses in-flight work#1113
aparajon merged 7 commits into
mainfrom
armand/poll-lost-work

aparajon commented Aug 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Kiran01bm commented Aug 28, 2026

Uh oh!

aparajon commented Aug 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

aparajon commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What breaks today

The fix

Also: the drive says when it's stuck

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Kiran01bm commented Aug 28, 2026

Non-blocking

General suggestions

The one thing that could have broken, verified

Verified correct

Uh oh!

aparajon commented Aug 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aparajon commented Aug 23, 2026 •

edited

Loading